Fttg Equatos Idea: Te varale of terest (depedet varale, y ) s ard to measure. Tere are easy to measure varales (predctor/ depedet) tat are related to te varale of terest, laeled,,... m We measure te y ad te s for a sample ad use ts sample to ft a model. Oce te model s ftted, we ca te just measure te s, ad get a estmate of y wtout measurg t Types of Equatos Smple Lear Equato: y β o + β + ε Multple Lear Equato: y β + β + β +...+β m m +ε Nolear Equato: takes may forms, for eample: Eample: Tree Hegt (m) ard to measure; D (dameter at. m aove groud cm) easy to measure use D squared for a lear equato e g t y y Dfferece etwee measured y ad te mea of y y yˆ. 6.. 8. 4.. 5.. 5.. 5.. 5. D squared ˆ Dfferece etwee measured y ad predcted y y y ˆ y y y ˆ ( y y) ( y yˆ ) y Dfferece etwee predcted y ad mea of y y y ˆ y ˆ y y β + β β β +ε
Ojectve: Fd estmates of β, β, β... β m suc tat te sum of squared dffereces etwee measured y ad predcted y (usually laeled as ŷ, values o te le or surface) s te smallest (mmze te sum of squared errors, called least squared error). OR Fd estmates of β, β, β... β m suc tat te lkelood (proalty) of gettg tese y values s te largest (mamze te lkelood). Fdg te mmum of sum of squared errors s ofte easer. I some cases, tey lead to te same estmates of parameters. Least Squares Soluto: Fdg te Set of Coeffcets tat Mmzes te Sum of Squared Errors To fd te estmated coeffcets tat mmzes SSE for a partcular set of sample data ad a partcular equato (form ad varales):. Defe te sum of squared errors (SSE) terms of te measured mus te predcted y s (te errors);. Take partal dervatves of te SSE equato wt respect to eac coeffcet. Set tese equal to zero (for te mmum) ad solve for all of te equatos (solve te set of equatos usg algera or lear algera). 4
Smple Lear Regresso Propertes of ad Tere s oly oe varale Tere wll e two coeffcets Te estmated tercept s foud y: ad are least squares estmates of β ad β. Uder assumptos cocerg te error term ad samplg/ measuremets, tese are: Uased estmates; gve may estmates of te slope ad tercept for all possle samples, te average of te sample estmates wll equal te true values y Ad te estmated slope s foud y: ( y y)( ) ( ) s s y ( ) ( ) SPy SS Te varalty of tese estmates from sample to sample ca e estmated from te sgle sample; tese estmated varaces wll e uased estmates of te true varaces (ad stadard errors) Te estmated tercept ad slope wll e te most precse (most effcet wt te lowest varaces) estmates possle (called Best ) Tese wll also e te mamum lkelood estmates of te tercept ad slope Were SPy refers to te corrected sum of cross products for ad y; SS refers to te corrected sum of squares for [Class eample] 5 6
Assumptos of SLR Oce coeffcets are otaed, we must ceck te assumptos of SLR. Assumptos must e met to: ota te desred caracterstcs assess goodess of ft (.e., ow well te regresso le fts te sample data) test sgfcace of te regresso ad oter ypoteses calculate cofdece tervals ad test ypotess for te true coeffcets (populato) calculate cofdece tervals for mea predcted y value gve a set of value (.e. for te predcted y gve a partcular value of te ) Need good estmates (uased or at least cosstet) of te stadard errors of coeffcets ad a kow proalty dstruto to test ypoteses ad calculate cofdece tervals. Ceckg te followg assumptos usg resdual Plots. a lear relatosp etwee te y ad te ;. equal varace of errors across te rage of te y varales; ad. depedece of errors (depedet oservatos), ot related tme or space. A resdual plot sows te resdual (.e., y - ŷ ) as te y-as ad te predcted value ( ŷ ) as te -as. Resdual plots ca also dcate uusual pots (outlers) tat may e measuremet errors, trascrpto errors, etc. 7 8
Eamples of Resdual Plots Idcatg Falures to Meet Assumptos:. Te relatosp etwee te s ad y s lear. If ot. Te varace of te y values must e te same for every oe of te values. If ot met, te spread aroud te le wll ot e eve. met, te resdual plot ad te plot of y vs. wll sow a curved le: [CRITICAL ASSUMPTION!!] t 6 ˆ 5 ˆ 4 ˆ ˆ 4 4 ˆ 4 4 44 ˆ 4 ˆ Šˆ------------ˆ------------ˆ------------ˆ------------ˆ------------ˆ 4 6 8 dsq Resdual 5 ˆ * * * * * * ˆ * * *** * *** *** *** ** ** * *** ***** *** *** * *** * 5 ˆ ********* ** ******* * **** * ** ** * ****** *** ** * ***** ** * ** *** * * * ***** ** * * * **** ˆ ******** ** *** ******** * * * * ** ******* * * ******* ** * * * ******* * * * -5 ˆ * *** * * * * * * ** ** * * ** * ** * * * ** ** - ˆ * * * * * * * -5 ˆ * * - ˆ Š-ˆ---------ˆ---------ˆ---------ˆ---------ˆ---------ˆ---------ˆ- 4 5 6 7 Predcted Value of t Result: If ts assumpto s ot met, te estmated coeffcets (slopes ad tercept) wll e uased, ut te estmates of te stadard devato of tese coeffcets wll e ased. we caot calculate CI or test te sgfcace of te Result: If ts assumpto s ot met: te regresso le does ot ft te data well; ased estmates of coeffcets varale. However, estmates of te coeffcets of te regresso le ad goodess of ft are stll uased ad stadard errors of te coeffcets wll occur 9
. Eac oservato (.e., ad y ) must e depedet of all oter oservatos. I ts case, we produce a dfferet resdual plot, were te resduals are o te y-as as efore, ut te -as s te varale tat s tougt to produce te depedeces (e.g., tme). If ot met, ts revsed resdual plot wll sow a tred, dcatg te resduals are ot depedet. Result: If ts assumpto s ot met, te estmated coeffcets (slopes ad tercept) wll e uased, ut te estmates of te stadard devato of tese coeffcets wll e ased. we caot calculate CI or test te sgfcace of te varale. However, estmates of te coeffcets of te regresso le ad goodess of ft are stll uased Normalty Hstogram or Plot A fourt assumpto of te SLR s: 4. Te y values must e ormally dstruted for eac of te values. A stogram of te errors, ad/or a ormalty plot ca e used to ceck ts, as well as tests of ormalty Hstogram # Boplot.5+*.*.*.*.**** 8.******* 4.************** 7.******************** 4.***************************** 57 +-----+.************************** 5.****************************** 6 *--+--* -.5+***************************** 58.************************* 49.***************** +-----+.************** 8.************ 4.***********.**** 7.**** 7.*** 5..* -.5+** ----+----+----+----+----+----+ HO: resduals are ormal H: resduals are ot ormal Tests for Normalty Test --Statstc--- -----p Value------ Sapro-Wlk W.99 Pr < W.9 Kolmogorov-Smrov D.98 Pr > D.67 Cramer-vo Mses W-Sq.96 Pr > W-Sq.66 Aderso-Darlg A-Sq.986 Pr > A-Sq <.5
Normal Proalty Plot.5+ * * +** +++** +**** +**** ***** **** ***** **** **** -.5+ **** ***+ **** *** +*** ***** +** +*** +**** * -.5+* +----+----+----+----+----+----+----+----+----+----+ Result: We caot calculate CI or test te sgfcace of te varale, sce we do ot kow wat proaltes to use. Also, estmated coeffcets are o loger equal to te mamum lkelood soluto. Resdual Frequecy 4 - - 5 - Normal Plot of Resduals - - - - Normal Score Hstogram of Resduals Resdual Volume versus d 4 Resdual Resdual 4 - - - 4 - - 5 56 6 6 I Cart of Resduals 6 6 7 7 6 5 5 Oservato Numer 5 Ft 7 7 5 Resduals vs. Fts 6 5 UCL.6 X. LCL-.6 4
Measuremets ad Samplg Assumptos Te remag assumptos are ased o te measuremets ad collecto of te samplg data. 5. Te values are measured wtout error (.e., te values are fed). Ts ca oly e kow f te process of collectg te data s kow. For eample, f tree dameters are very precsely measured, tere wll e lttle error. If ts assumpto s ot met, te estmated coeffcets (slopes ad tercept) ad ter varaces wll e ased, sce te values are varyg. 6. Te y values are radomly selected for value of te varales (.e., for eac value, a lst of all possle y values s made, ad some are radomly selected). Ofte, te oservatos wll e gatered usg systematc samplg (grd across te lad area). Ts does ot strctly meet ts assumpto. Also, more comple samplg desg suc as multstage samplg (samplg large uts ad samplg smaller uts wt te large uts), ts assumpto s ot met. If te equato s correct, te ts does ot cause prolems. If ot, te estmated equato wll e ased. Trasformatos Commo Trasformatos Powers,.5, etc. for relatosps tat look olear log, loge also for relatosps tat look olear, or we te varaces of y are ot equal aroud te le S- [arcse] we te depedet varale s a proporto. Rak trasformato: for o-ormal data o Sort te y varale o Assg a rak to eac varale from to o Trasform te rak to ormal (e.g., Blom Trasformato) PROBLEM: loose some of te formato te orgal data Try to trasform frst ad leave y varale of terest; owever, ts s ot always possle. Use graps to elp coose trasformatos 5 6
Outlers: Uusual Pots Ceck for pots tat are qute dfferet from te oters o: Grap of y versus Resdual plot Do ot delete te pot as t MAY BE VALID! Ceck: Is ts a measuremet error? E.g., a tree egt of m s very ulkely Is a trascrpto error? E.g. for adult perso, a wegt of ls was etered rater ta ls. Is tere sometg very uusual aout ts pot? e.g., a rd as a sort eak, ecause t was damaged. Try to f te oservato. If t s very dfferet ta te oters, or you kow tere s a measuremet error tat caot e fed, te delete t ad dcate ts your researc report. O te resdual plot, a outler CAN occur f te model s ot correct may eed a trasformato of te varale(s), or a mportat varale s mssg Measures of Goodess of Ft How well does te regresso ft te sample data? For smple lear regresso, a grap of te orgal data wt te ftted le marked o te grap dcates ow well te le fts te data [ot possle wt MLR] Two measures commoly used: coeffcet of determato (r ) ad stadard error of te estmate(se E ). To calculate r ad SE E, frst, calculate te SSE (ts s wat was mmzed): SSE e ( y yˆ ) ( y ( + )) Te sum of squared dffereces etwee te measured ad estmated y s. Calculate te sum of squares for y: ( ) SSy y y y y sy ( ) Te sum of squared dfferece etwee te measured y ad te mea of y-measures. NOTE: I some tets, ts s called te sum of squares total. 7 8
Calculate te sum of squares regresso: SSreg ( y yˆ ) SPy SSy SSE Te sum of squared dffereces etwee te mea of y- measures ad te predcted y s from te ftted equato. Also, s te sum of squares for y te sum of squared errors. SSy SSE SSE SSreg Te: r SSy SSy SSy SSE, SSY are ased o y s used te equato wll ot e orgal uts f y was trasformed r coeffcet of determato; proporto of varace of y, accouted for y te regresso usg Is te square of te correlato etwee ad y O (very poor orzotal surface represetg o relatosp etwee y ad s) to (perfect ft surface passes troug te data) Ad: SSE SE E SSE s ased o y s used te equato wll ot e orgal uts f y was trasformed SE E - stadard error of te estmate; same uts as y Uder ormalty of te errors: o ± SE E 68% of sample oservatos o ± SE E 95% of sample oservatos o Wat low SEE 9
y-varale was trasformed: Ca calculate estmates of tese for te orgal y-varale ut, called I (Ft Ide) ad estmated stadard error of te estmate (SE E ), order to compare to r ad SE E of oter equatos were te y was ot trasformed. I - SSE/SSY were SSE, SSY are orgal uts. NOTE must ack-trasform te predcted y s to calculate te SSE orgal uts. Does ot ave te same propertes as r, owever: o t ca e less ta o t s ot te square of te correlato etwee te y ( orgal uts) ad te used te equato. Estmated stadard error of te estmate (SE E ), we te depedet varale, y, as ee trasformed: SSE( orgal uts) SE E ' SE E - stadard error of te estmate ; same uts as orgal uts for te depedet varale wat low SE E [Class eample] Estmated Varaces, Cofdece Itervals ad Hypotess Tests Testg Weter te Regresso s Sgfcat Does kowledge of mprove te estmate of te mea of y? Or s t a flat surface, wc meas we sould just use te mea of y as a estmate of mea y for ay? SSE/ (-): Called te Mea squared error, as would e te average of te squared error f we dvded y. Istead, we dvde y -. Wy? Te degrees of freedom are -; oservatos wt two statstcs estmated from tese, ad Uder te assumptos of SLR, s a uased estmated of te true varace of te error terms (error varace) SSR/: Called te Mea Square Regresso Degrees of Freedom: -varale Uder te assumptos of SLR, ts s a estmate te error varace PLUS a term of varace eplaed y te regresso usg.
H: Regresso s ot sgfcat H: Regresso s sgfcat Same as: H: β [true slope s zero meag o relatosp wt ] H: β [slope s postve or egatve, ot zero] Ts ca e tested usg a F-test, as t s te rato of two varaces, or wt a t-test sce we are oly testg oe coeffcet (more o ts later) Iformato for te F-test s ofte sow as a Aalyss of Varace Tale: Source df SS MS F p-value Regresso MSreg F Pro F> SSreg SSreg/ MSreg/MSE F (,-,- α) Resdual - SSE MSE SSE/(-) Total - SSy [Class eample ad eplaato of te p-value] Usg a F test statstc: SSreg F SSE ( ) MSreg MSE Uder H, ts follows a F dstruto for a - α/ percetle wt ad - degrees of freedom. If te F for te ftted equato s larger ta te F from te tale, we reject H (ot lkely true). Te regresso s sgfcat, tat te true slope s lkely ot equal to zero. 4
Estmated Stadard Errors for te Slope ad Itercept Uder te assumptos, we ca ota a uased estmated of te stadard errors for te slope ad for te tercept [measure of ow tese would vary amog dfferet sample sets], usg te oe set of sample data. s s MSE + SS MSE SS MSE SS Cofdece Itervals for te True Slope ad Itercept Uder te assumptos, cofdece tervals ca e calculated as: For β o : ± t α, s Hypotess Tests for te True Slope ad Itercept H: β c [true slope s equal to te costat, c] H: β c [true slope dffers from te costat c] Test statstc: t c s Uder H, ts s dstruted as a t value of t c t -, -α/. Reject H o f t > t c. Te procedure s smlar for testg te true tercept for a partcular value It s possle to do oe-sded ypoteses also, were te alteratve s tat te true parameter (slope or tercept) s greater ta (or less ta) a specfed costat c. MUST e careful wt te t c as ts s dfferet. [class eample] t s ± For β : α, [class eample] 5 6
Cofdece Iterval for te True Mea of y gve a partcular value For te mea of all possle y-values gve a partcular value of (μ y ): were yˆ s yˆ ˆ y ± t, α syˆ + MSE + ( ) SS Cofdece Bads Plot of te cofdece tervals for te mea of y for several -values. Wll appear as:. 8. 6. 4... 8. 6. 4... 5.. 5.. 5.. 5. Predcto Iterval for or more y-values gve a partcular value For oe possle ew y-value gve a partcular value of : Were yˆ s ˆ y ( ew) ± t, α syˆ ( ew) ( ew) yˆ ( ew) + MSE + + ( ) SS For te average of g ew possle y-values gve a partcular value of : were yˆ s ( ew) ˆ y ( ew) ± t, α syˆ ( ewg ) yˆ ( ew g ) [class eample] + MSE + + g ( ) SS 7 8
Selectg Amog Alteratve Models Process to Ft a Equato usg Least Squares Steps:. Sample data are eeded, o wc te depedet varale ad all eplaatory (depedet) varales are measured.. Make ay trasformatos tat are eeded to meet te most crtcal assumpto: Te relatosp etwee y ad s lear. Eample: volume β + β d may e lear wereas volume versus d s ot. Use y volume, d.. Ft te equato to mmze te sum of squared error. 4. Ceck Assumptos. If ot met, go ack to Step. 5. If assumptos are met, te terpret te results. Is te regresso sgfcat? Wat s te r? Wat s te SE E? Plot te ftted equato over te plot of y versus. For a umer of models, select ased o:. Meetg assumptos: If a equato does ot meet te assumpto of a lear relatosp, t s ot a caddate model. Compare te ft statstcs. Select ger r (or I ), ad lower SE E (or SE E ). Reject ay models were te regresso s ot sgfcat, sce ts model s o etter ta just usg te mea of y as te predcted value. 4. Select a model tat s ologcally tractale. A smpler model s geerally preferred, uless tere are practcal/ologcal reasos to select te more comple model 5. Cosder te cost of usg te model [class eample] 9
Smple Lear Regresso Eample wegt versus temperature Temperature () Wegt (y) Wegt (y) 8 6 8 5 4 5 4 45 8 6 44 9 4 75 48 5 44 Oservato temp wegt 8 6 8 4 5 5 5 6 5 4 7 5 8 Et cetera Wegt (y) wegt 6 5 4 4 5 6 7 8 temperature
Os. temp wegt -dff -dff. sq. 8-7.5 46.5 6-7.5 46.5 8-7.5 46.5 4 5 -.5 56.5 Et cetera mea 7.5 7. SSX,8.5 SSY,9.8 SPXY6,75. SPy y SS :.56769 : 5.8597 NOTE: calculate frst, sce ts s eeded to calculate. From tese, te resduals (errors) for te equato, ad te sum of squared error (SSE) were calculated: Os. wegt y-pred resdual resdual sq. 8 5.8.7 4.7 6 5.8.7. 8 5.8.7 4.7 4 4.4 -.4 5.47 Et cetera SSE: 5.89 Ad SSRSSY-SSE85.89 ANOVA Source df SS MS Model 85.89 85.89 Error 8-6 5.89 6.6 Total 8-7 9.78 4
F575.6 wt p. (very small) I ecel use: fdst(,df,df) to ota a p-value r :.97 Root MSE Or SE E :.57 BUT: Before terpretg te ANOVA tale, Are assumptos met? If assumptos were ot met, we would ave to make some trasformatos ad start over aga! resduals ( erro rs) 6. 4... -. -4. -6. Lear? resdual plot.... 4. 5. 6. Equal varace? predcted wegt Idepedet oservatos? [eed aoter plot resduals versus tme or space, tat cause depedeces] 5 6
Normalty plot: Os. sorted Stad. Rel. Pro. resds resds Freq. z- dst. -4.4 -.7.6.4-4.4 -.69..5 -.7 -..7. 4 -.4 -.9..8 5 -.85 -.7.8.4 6 -.88 -.4..7 7 -.4 -.5.9.44 8 -.7 -.4.44.44 9 -.4 -..5.45 Etc. cum ulatve proalty...8.6.4.. Questos: Proalty plot -. -.... z-value. Are te assumptos of smple lear regresso met? Evdece? relatve frequecy Pro. z-dst.. If so, terpret f ts s a good equato ased o goodess of t measures.. Is te regresso sgfcat? 7 8
For 95% cofdece tervals for ad, would also eed estmated stadard errors: s s MSE + SS MSE SS 6.6 8.5 6.6.7 8 7.5 +.75 8.5 Te t-value for 6 degrees of freedom ad te.975 percetle s. (tv(.5,6) EXCEL) ± t α, s For β o : 5.85 ±.. 75 t ± α, s For β :.568 ±.. 7 Est. Coeff St. Error For : 5.859685.7497559 For :.5676948.679 CI: t(.975,6).. lower.5464588.57485 upper 8.4477.6779974 Questo: Could te real tercept e equal to? Gve a temperature of, wat s te estmated average wegt (predcted value) ad a 95% cofdece terval for ts estmate? 9 4
yˆ yˆ ( s s yˆ yˆ + ) 5.85 +.568 8. MSE + 6.6 8 ( ) + SS ( 7.5) 8.5.79 Gve a temperature of, wat s te estmated wegt for ay ew oservato, ad a 95% cofdece terval for ts estmate? yˆ yˆ ( + ) 5.85 +.568 8. yˆ ± t, α syˆ 8...79 8. +..79 6.8 9.86 s s yˆ yˆ MSE + + 6.6 + 8 ( ) + SS ( 7.5) 8.5.669 yˆ ± t, α syˆ 8...669 8. +..669.66.97 4 4
Multple Lear Regresso (MLR) For eample: Populato: y β + β + β +...+β p m +ε Sample: y + + +...+ p m +e yˆ + + + K + m m e y β o s te y tercept parameter β, β, β,..., β m are slope parameters,,... m depedet varales ε - s te error term or resdual - s te varato te depedet varale (te y) wc s ot accouted for y te depedet varales (te s). For ay ftted equato (we ave te estmated parameters), we ca get te estmated average for te depedet varale, for ay set of s. Ts wll e te predcted value for y, wc s te estmated average of y, gve te partcular values for te varales. NOTE: I tet y Neter et al. pm+. Ts s ot e cofused wt te p- value dcatg sgfcace ypotess tests. yˆ Predcted log(vol) - 4. +. X log(d) +. X log(egt) were o -4.;. ;. estmated y fdg te least squared error soluto. Usg ts equato for d cm, egt8m, logte(d).48, logte(egt).45; logte(vol).5. volume (m ).84. Ts represets te estmated average volume for trees wt d cm ad egt8 m. Note: Ts equato s orgally a olear equato: vol a d Wc was trasformed to a lear equato usg logartms: log ( vol) log( a) + log( d) + c log( t) + logε Ad ts was ftted usg multple lear regresso t c ε 4 44
For te oservatos te sample data used to ft te regresso, we ca also get a estmate of te error (we ave measured volume). If te measured volume for ts tree was. m, or.477 log uts: error y yˆ.477.5.6 For te ftted equato usg log uts. I orgal uts, te estmated error s.-.84 -.84 NOTE: Ts s ot smply te atlog of -.6. Fdg te Set of Coeffcets tat Mmzes te Sum of Squared Errors Same process as for SLR: Fd te set of coeffcets tat results te mmum SSE, just tat tere are more parameters, terefore more partal dervatve equatos ad more equatos o E.g., wt -varales, tere wll e 4 coeffcets (tercept plus slopes) so four equatos For lear models, tere wll e oe uque matematcal soluto. For olear models, ts s ot possle ad we must searc to fd a soluto Usg te crtero of fdg te mamum lkelood (proalty) rater ta te mmum SSE, we would eed to searc for a soluto, eve for lear models (covered oter courses, e.g., FRST 5). 45 46
47 Least Squares Metod for MLR: Fd te set of estmated parameters (coeffcets) tat mmze sum of squared errors ( ) + + + + m p y e SSE )... ( m ) m( ) m( Take partal dervatves wt respect to eac of te coeffcets, set tem equal to zero ad solve. For tree -varales we ota: y SS SP SS SP SS y SP SS SP SS SP SS y SP SS SP SS SP SS y SP 48 Were SP dcates sum of products etwee two varales, for eample for y wt : ( )( ) ) ( s y y y y y SP y Ad SS dcates sums of squares for oe varale, for eample for : ( ) ) ( s SS
Propertes of a least squares regresso surface :. Always passes troug (,,,..., m, y). Sum of resduals s zero,.e., Σe. SSE te least possle (least squares) 4. Te slope for a partcular -varale s AFFECTED y correlato wt oter -varales: CANNOT terpret te slope for a partcular -varale, UNLESS t as zero correlato wt all oter - varales (or early zero f correlato s estmated from a sample). [class eample] Meetg Assumptos of MLR Oce coeffcets are otaed, we must ceck te assumptos of MLR efore we ca: assess goodess of ft (.e., ow well te regresso le fts te sample data) test sgfcace of te regresso calculate cofdece tervals ad test ypotess For tese test to e vald, assumptos of MLR cocerg te oservatos ad te errors (resduals) must e met. 49 5
Resdual Plots Assumptos of:. Te relatosp etwee te s ad y s lear VERY IMPORTANT!. Te varaces of te y values must e te same for every comato of te values.. Eac oservato (.e., s ad y ) must e depedet of all oter oservatos. ca e vsually cecked y usg RESIDUAL PLOTS Normalty Hstogram or Plot A fourt assumpto of te MLR s: 4. Te y values must e ormally dstruted for eac comato of values. A stogram of te errors, ad/or a ormalty plot ca e used to ceck ts, as well as tests of ormalty as wt SLR. Falure to meet tese assumptos wll result same prolems as wt SLR. A resdual plot sows te resdual (.e., y - ŷ ) as te y-as ad te predcted value ( ŷ ) as te -as. For te depece assumpto, te -as s tme or space tat eplas te depedece of te data. THIS IS THE SAME as for SLR. Look for prolems as wt SLR. Te effects of falg to meet a partcular assumpto are te same as for SLR Wat s dfferet? Sce tere are may varales, t wll e arder to decde wat to do to f ay prolems. 5 5
Eample: Lear relatosp met, equal varace, o evdece of tred wt oservato umer (depedece may e met). Also, ormal dstruto met. Logvolf(d,logd) R e s d u a l F requecy.. -. -. 9 8 7 6 5 4 Normal Plot of Resduals - - - Normal Score Resdual Model Dagostcs Hstogram of Resduals -.-.5-.-.5..5..5 Resdual R e s d u a l R e s d u a l..5..5. -.5 -. -.5 -. -.5.. -. -. I Cart of Resduals 5-5 5 5 Oservato Numer Resduals vs. Fts Ft X. 5 UCL.78 LCL-.78 5 Lear relatosp assumpto ot met Resdual Frequecy 4 - - 5 - Normal Plot of Resduals - - - - Normal Score Hstogram of Resduals Resdual Volume versus d 4 Resdual Resdual 4 - - - 4 - - 5 56 6 6 I Cart of Resduals 6 6 7 7 6 5 5 Oservato Numer 5 Ft 7 7 5 Resduals vs. Fts 6 5 UCL.6 X. LCL-.6 54
Resdual Varaces are ot equal Frequecy.5..5..5. -.5 -. -.5 -. 5 5 - Volume versus d squared ad d Normal Plot of Resduals - - Normal Score Hstogram of Resduals -.-.5-.-.5..5..5..5 Resdual Resdual Resdual - -.5..5..5. -.5 -. -.5 -. I Cart of Resduals 777 7 7 7 77 7 7 7 5 5 Oservato Numer 5 Ft 5 5 Resduals vs. Fts X. 5 5 UCL.68 LCL-.68 55 Measuremets ad Samplg Assumptos Te remag assumptos of MLR are ased o te measuremets ad collecto of te samplg data, as wt SLR 5. Te values are measured wtout error (.e., te values are fed). 6. Te y values are radomly selected for eac gve set of te varales (.e., for eac fed set of values, a lst of all possle y values s made). As wt SLR, ofte oservatos wll e gatered usg smple radom samplg or systematc samplg (grd across te lad area). Ts does ot strctly meet ts assumpto [muc more dffcult to meet wt may - varales!] If te equato s correct, te ts does ot cause prolems. If ot, te estmated equato wll e ased. 56
Trasformatos Measures of Goodess of Ft Same as for SLR ecept tat tere are more varales; ca also add varales e.g. use d ad d as ad. Try to trasform s frst ad leave y varale of terest; ot always possle. Use graps to elp coose trasformatos Wll result a teratve process:. Ft te equato. Ceck te assumptos [ad ceck for outlers]. Make ay trasformatos ased o te resdual plot, ad plots of y versus eac 4. Also, ceck ay very uusual pots to see f tese are measuremet/trascrpto errors; ONLY remove te oservato f tere s a very good reaso to do so 5. Ft te equato aga, ad ceck te assumptos 6. Cotue utl te assumptos are met [or early met] How well does te regresso ft te sample data? For multple lear regresso, a grap of te te predcted versus measured y values dcates ow well te le fts te data Two measures commoly used: coeffcet of multple determato (R ) ad stadard error of te estmate(se E ), smlar to SLR To calculate R ad SE E, frst, calculate te SSE (ts s wat was mmzed, as wt SLR): SSE ( y yˆ ) ( y ( + + +... m m )) e Te sum of squared dffereces etwee te measured ad estmated y s. Ts s te same as for SLR, ut tere are more slopes ad more (predctor) varales. 57 58
Calculate te sum of squares for y: ( ) SSy y y y y sy ( ) Te sum of squared dfferece etwee te measured y ad te mea of y-measures. Calculate te sum of squares regresso: SSreg ( y yˆ ) SSy SSE SP y + SP y +... + SP y Te sum of squared dffereces etwee te mea of y- measures ad te predcted y s from te ftted equato. Also, s te sum of squares for y te sum of squared errors. SSy SSE SSE SSreg Te: R SSy SSy SSy SSE, SSY are ased o y s used te equato wll ot e orgal uts f y was trasformed R coeffcet of multple determato; proporto of varace of y, accouted for y te regresso usg s O (very poor orzotal surface represetg o relatosp etwee y ad s) to (perfect ft surface passes troug te data) SSE falls as m (umer of depedet varale) creases, so R rses as more eplaatory (depedet or predctor) varales are added. A smlar measure s called te Adjusted R value. A pealty s added as you add -varales to te equato: R a SSE m ( + ) SSy 59 6
SSE Ad: SE E m SSE s ased o y s used te equato wll ot e orgal uts f y was trasformed -m- s te degrees of freedom for te error; s te umer of oservatos mus te umer of ftted coeffcets SE E - stadard error of te estmate; same uts as y Uder ormalty of te errors: o ± SE E 68% of sample oservatos o ± SE E 95% of sample oservatos Wat low SE E SE E falls as te umer of predctor varales creases ad SSE falls, ut te rses, sce -m - s gettg smaller y-varale was trasformed: Ca calculate estmates of tese for te orgal y-varale ut, I (Ft Ide) ad estmated stadard error of te estmate (SE E ), order to compare to R ad SE E of oter equatos were te y was ot trasformed, smlar to SLR. I - SSE/SSY were SSE, SSY are orgal uts. NOTE must ack-trasform te predcted y s to calculate te SSE orgal uts. Does ot ave te same propertes as R, owever t ca e less ta Estmated stadard error of te estmate (SE E ), we te depedet varale, y, as ee trasformed: SSE( orgal uts) SE E ' m SEE - stadard error of te estmate ; same uts as orgal uts for te depedet varale wat low SEE 6 6
Estmated Varaces, Cofdece Itervals ad Hypotess Tests Testg Weter te Regresso s Sgfcat Does kowledge of s mprove te estmate of te mea of y? Or s t a flat surface, wc meas we sould just use te mea of y as a estmate of mea y for ay set of values? SSE/ (-m-): Mea squared error. o Te degrees of freedom are -m- (same as -(m+) o oservatos wt (m+) statstcs estmated from tese:,,, m Uder te assumptos of MLR, s a uased estmated of te true varace of te error terms (error varace) SSR/m: Called te Mea Square Regresso Degrees of Freedomm: m -varales Uder te assumptos of MLR, ts s a estmate te error varace PLUS a term of varace eplaed y te regresso usg s. H: Regresso s ot sgfcat H: Regresso s sgfcat Same as: H: β β β... β m [all slopes are zero meag o relatosp wt s] H: ot all slopes [some or all slopes are ot equal to zero] If H s true, te te equato s: y β + + +...+ m +ε y β ˆ + ε y β Were te -varales ave o fluece over y; tey do ot elp to etter estmate y. 6 64
As wt SLR, we ca use a F-test, as t s te rato of two varaces; ulke SLR we caot use a t-test sce we are oly testg several slope coeffcets. Usg a F test statstc: SSreg m F SSE ( m ) MSreg MSE Uder H, ts follows a F dstruto for a - α percetle wt m ad -m- degrees of freedom. If te F for te ftted equato s larger ta te F from te tale, we reject H (ot lkely true). Te regresso s sgfcat, tat oe or more of te te true slopes (te populato slopes) are lkely ot equal to zero. Iformato for te F-test te Aalyss of Varace Tale: Source df SS MS F p-value Regresso m MSreg F Pro F> SSreg SSreg/m MSreg/MSE F (m,-m-,- α) Error -m- SSE MSE SSE/( m-) Estmated Stadard Errors for te Slope ad Itercept Uder te assumptos, we ca ota a uased estmated of te stadard errors for te slope ad for te tercept [measure of ow tese would vary amog dfferet sample sets], usg te oe set of sample data. For multple lear regresso, tese are more easly calculated usg matr algera. If tere are more ta - varales, te calculatos ecome dffcult; we wll rely o statstcal packages to do tese calculatos. Cofdece Itervals for te True Slope ad Itercept Uder te assumptos, cofdece tervals ca e calculated as: For β o : ± t α, m s For β j : j ± t α, m s [ for ay of te slopes] j Total - SSy [See eample] [See eample] 65 66
Hypotess Tests for oe of te True Slopes or Itercept H: β j c [te parameter (true tercept or true slope s equal to te costat, c, gve tat te oter -varales are te equato] H: β j c [true tercept or slope dffers from te costat c; gve tat te oter -varales are te equato] Te regresso s sgfcat, ut wc -varales sould we reta? Wt MLR, we are partcularly terested wc - varales to reta. We te test: Is varale j sgfcat gve te oter varales? e.g. dameter, egt - do we eed ot? Test statstc: j c t s Uder H, ts s dstruted as a t value of t c t -m-, -α/. Reject H o f t > t c. It s possle to do oe-sded ypoteses also, were te alteratve s tat te true parameter (slope or tercept) s greater ta (or less ta) a specfed costat c. MUST e careful wt te t c as ts s dfferet. [See eample] j H: β j, gve oter -varales (.e., varale ot sgfcat) H: β j, gve oter -varales. A t-test for tat varale ca e used to test ts. 67 68
Aoter test, te partal F-test ca e used to test oe - varale (as t-test) or to test a group of -varales, gve te oter -varales te equato. Get regresso aalyss results for all -varales [full model] Get regresso aalyss results for all ut te -varales to e tested [reduced model] partal F OR partal F ( SSreg( full) SSreg( reduced) ) SSE ( m )( full) ( SSE( reduced) SSE( full) ) SSE ( m )( full) ( SS due to dropped varale(s)) /r MSE( full) Were r s te umer of -varales tat were dropped (also equals: ()te regresso degrees of freedom for te full model mus te regresso degrees of freedom for te reduced model, OR () te error degrees of freedom for te reduced model, mus te error degrees of freedom for te full model) r r Uder H, ts follows a F dstruto for a - α percetle wt r ad -m- (full model) degrees of freedom. If te F for te ftted equato s larger ta te F from te tale, we reject H (ot lkely true). Te regresso s sgfcat, tat te varale(s) tat were dropped are sgfcat (accout for varace of te y-varale), gve tat te oter -varales are te model. [See eample wt te use of class varales, ut ca e for ay suset of -varales] 69 7
Cofdece Iterval for te True Mea of y gve a partcular set of values For te mea of all possle y-values gve a partcular value set of -values (μ y ): were yˆ yˆ ± t m, α syˆ + + + L+ m m yˆ s yˆ ( ew) + + + L+ m from statstcal package output For te average of g ew possle y-values gve a partcular value of : were yˆ yˆ ( ew) ± t m, α syˆ ( ewg ) + + m + L+ m m s yˆ from statstcal package output Cofdece Bads Plot of te cofdece tervals for te mea of y for several sets -values s ot possle wt MLR s yˆ ( ewg) [See eample] from statstcal package output Predcto Iterval for or more y-values gve a partcular set of values For oe possle ew y-value gve a partcular set of values: Were yˆ ( ew) ± t m, α syˆ ( ew) 7 7
Selectg ad Comparg Alteratve Models Process to Ft a Equato usg Least Squares Steps (same as for SLR):. Sample data are eeded, o wc te depedet varale ad all eplaatory (depedet) varales are measured.. Make ay trasformatos tat are eeded to meet te most crtcal assumpto: Te relatosp etwee y ad s s lear. Eample: volume β + β d +β d may e lear wereas volume versus d s ot. Need ot varales.. Ft te equato to mmze te sum of squared error. 4. Ceck Assumptos. If ot met, go ack to Step. 5. If assumptos are met, te ceck f te regresso s sgfcat. If t s ot, te t s ot a caddate model (eed oter -varales). If yes, te go troug furter steps for MLR. 6. Are all varales eeded? If tere are -varales tat are ot sgfcat, gve te oter varales: drop te least sgfcat oe (gest p-value, or lowest asolute value of t) reft te regresso ad ceck assumptos. f assumptos are met, te repeat steps 5 ad 6 cotue utl all varales te regresso are sgfcat gve te oter -varales also te model 7 74
Metods to ad selectg predctor () varales Metods ave ee developed to elp coosg wc - varales to clude te equato. Tese clude:. Forward: Brg varales oe at a tme, utl te remag oes are o loger sgfcat, gve te oters already te equato. ( oly). Backward: Drop varales oe at a tme, utl all remag varales are sgfcat, gve te oters stll te equato (out oly). Stepwse ( ad out) NOTE: Tese tools just gves caddate models. You must ceck weter te assumptos are met ad do a full assessmet of te regresso results Steps for Forward Stepwse, for eample: To ft ts y ad, you would eed to do te followg steps:. Ft a smple lear regresso for vol/a wt eac of te eplaatory () varales.. Of te equatos tat are sgfcat (assumptos met?), select te oe wt te gest F-value.. Ft a MLR wt vol/a usg te selected varale, plus eac of te eplaatory varales ( -varales eac equatos). Ceck to see f te ew varale s sgfcat gve te orgal varale (wc may ow e ot sgfcat, ut forward stepwse does ot drop varales). Of te oes tat are sgfcat (gve te orgal varale s also te equato), pck te oe wt te largest partal-f (for te ew varale). 4. Repeat step, rgg varales utl ) tere are o more varales or ) te remag varales are ot sgfcat gve te oter varales. 75 76
SAS Outputs: forward stepwse Te REG Procedure Numer of Oservatos Read 8 Numer of Oservatos Used 8 Te REG Procedure Model: MODEL Depedet Varale: vola vola Numer of Oservatos Read 8 Numer of Oservatos Used 8 Forward Selecto: Step Descrptve Statstcs Ucorrected Varale Sum Mea SS Varace Itercept 8.. 8. aa.6 6.5574 944 74.9884 stemsa 6586 5.574 577 6984 qd 476.8 7.857 984. 5.7444 age 8. 7.4857 7697 4.84 s 4.9 5.57 6594.9 7.6658 topt 549.6 9.6857 8.66878 vola 4995 55.599 968 64 Descrptve Statstcs Stadard Varale Devato Lael Itercept Itercept aa 8.6567 aa stemsa 47.848 stemsa qd 5.97866 qd age.855 age s.7684 s topt.9448 topt vola 9.688 vola Varale aa Etered: R-Square.77 ad C(p) 87.5 Aalyss of Varace Sum of Mea Source DF Squares Square F Value Pr > F Model 75684 75684 87.7 <. Error 6 47 869.6976 Corrected Total 7 984 Parameter Stadard Type II F Varale Estmate Error SS Value Pr>F Itercept -7.4967 77.5 44 4.9.59 aa 9.449.65 75684 87.7 <. Bouds o codto umer:, 77 78
Forward Selecto: Step Varale topt Etered: R-Square.985 ad C(p) 4.549 Aalyss of Varace Sum of Mea Source DF Squares Square F Value Pr > F Model 96676 4868 84.64 <. Error 5 4478 579.68 Corrected Total 7 984 Parameter Stadard Type F Varale Estmate Error II SS Value Pr> F Itercept -66.989.796 8 4.9 <. aa 5.7874.56747 445489 769. <. topt.767.66846 9894 6.4 <. Bouds o codto umer:.5, 4.5 Forward Selecto: Step Varale stemsa Etered: R-Square.9879 ad C(p).6949 Aalyss of Varace Sum of Mea Source DF Squares Square F Value Pr > F Model 9698 7 655.5 <. Error 4 84 49.68 Corrected Total 7 984 Parameter Stadard Type Varale Estmate Error II SS F Value Pr> F Itercept-57.86686 6.85 77 75.6 <. aa 6.7897.599 779 765.4 <. stemsa -.9.569 644.9 5.6.94 topt 5.769.468 6 7. <. Bouds o codto umer: 4.4, 8.766 -------------------------------------------------- No oter varale met te.5 sgfcace level for etry to te model. 79 8
Summary of Forward Selecto Step Numer Partal Model Vars I R-Square R-Square C(p) F Value aa.77.77 87.5 87.7 topt.9.985 4.549 6.4 stemsa.7.9879.6949 5.6 Summary of Forward Selecto Step Pr > F <. <..94 For a umer of models, select ased o:. Meetg assumptos: If a equato does ot meet te assumpto of a lear relatosp, t s ot a caddate model. Compare te ft statstcs. Select ger R (or I ), ad lower SE E (or SE E ). Reject ay models were te regresso s ot sgfcat, sce ts model s o etter ta just usg te mea of y as te predcted value. 4. Select a model tat s ologcally tractale. A smpler model s geerally preferred, uless tere are practcal/ologcal reasos to select te more comple model 5. Cosder te cost of usg te model 8 8
Addg class varales as predctors Wat to add a class varale. Eamples:. Add speces to a equato to estmate tree egt.. Add geder (male/female) to a equato to estmate wegt of adult taled frogs.. Add mace type to a equato tat predcts lumer output. How s ts doe? Use dummy or dcator varales to represet te class varale e.g. ave speces. Set up X ad X as dummy varales: Speces X X Cedar Hemlock Douglas fr Oly eed two dummy varales to represet te tree speces. Te two dummy varales as a group represet te speces. Add te dummy varales to te equato ts wll alter te tercept To alter te slopes, add a teracto etwee dummy varales ad cotuous varale(s) e.g. ave speces, ad a cotuous varale, d Speces X X d X4X * d X5X*d Cedar Hemlock Douglas fr 5 NOTE: Tere would e more ta oe le of data (sample) for eac speces. o Te two dummy varales, ad te teractos wt te cotuous varale as a group represet te speces. 8 84
How does ts work? y + + + { + 4 4 + 5 5 + e 44 4 44 4 4 For Cedar (CW): dummy varales d teractos Oter metods, ta SLR ad Multple Lear Regresso, we trasformatos do ot work: Nolear least squares: Least squares soluto for olear models; uses a searc algortm to fd estmated coeffcets; as good propertes for large datasets; stll assumes ormalty, equal varaces, ad depedet oservatos For Hemlock (HW): Wegted least squares: for uequal varaces. Estmate te varaces ad use tese wegtg te least squares ft of te regresso; assumes ormalty ad depedet oservatos For Douglas fr (FD): Geearlzed lear model: used for dstrutos oter ta ormal (e.g., omal, Posso, etc.), ut wt o correlato etwee oservatos; uses mamum lkelood Terefore: ft oe equato usg all data, ut get dfferet equatos for dfferet speces. Also, ca test for dffereces amog speces, usg a partal-f test. Geeralzed least Squares ad Med Models: use mamum lkelood for fttg models wt uequal varaces, correlatos over space, correlatos over tme, ut ormally dstruted errors Geeralzed lear med models: Allows for uequal varaces, correlatos over space ad/or tme, ad o-ormal dstrutos; uses mamum lkelood 85 86